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ABSTRACT 

We  define  native  intelligence  as  the  specified  complexity 
inherent  in  the  information  content  of  an  artificial  system.  The 
artificial  system  is  defined  as  a  system  that  can  be  encoded  in 
some  general  purpose  language,  expressed  minimally  as  some 
finite  length  bit  string,  and  decoded  by  a  finite  set  of  rules 
defined  a  priori.  Using  this  definition  of  native  intelligence,  we 
employ  a  chance  elimination  argument  in  the  literature  to  form  a 
simple,  but  promising  native  intelligence  metric.  Several 
anticipated  objections  to  this  native  intelligence  metric  are 
discussed. 

1  Introduction 

We  define  two  perspectives  on  artificial  system 
intelligence:  (1)  native  intelligence,  expressed  in  the 
specified  complexity  inherent  in  the  information  content  of 
the  system,  and  (2)  performance  intelligence,  expressed  in 
the  successful  ( i.e .,  goal-achieving)  performance  of  the 
system  in  a  complicated  environment.  In  this  context, 
complexity  is  simply  Shannon  information  [5],  Specified 
complexity  is  Shannon  information  matched  with 
meaningful  patterns  (e.g.,  orthography,  syntax,  and 
semantics).  Native  intelligence  aggregates  things  like 
potential  intelligence,  learning  ability,  system  integration, 
richness  and  potential  effectiveness  of  individual 
behaviors,  and  intellectual  reasoning  capabilities.  These 
are  fundamental,  theoretical,  and  innate  aspects  of 
intelligence.  The  performance  perspective  on  intelligence 
aggregates  things  like  the  efficiency  of  design  and 
maintenance,  real-time  performance,  and  the  ability  to 
effect  desired  physical  changes  on  the  environment.  These 
are  external  behavior  characteristics  that  can  be  measured 
without  knowing  where  the  intelligence  came  from,  how  it 
is  represented,  or  what  algorithms  are  used  internally  to 
process  the  data  and  produce  effective  output. 

These  two  perspectives  have  a  parallel  with  the 
distinction  between  theoretical  and  experimental  sciences. 
In  this  regard,  the  native  and  performance  perspectives  are 


complimentary  and  not  competitive.  They  also  ought  to 
give  the  same  results  when  applied  to  the  same  quantities 
in  identical  systems.  Each  one  ought  to  inform  and 
contribute  insight  to  the  other. 

The  essence  of  intelligence  is  completely  non¬ 
material.  This  must  be  true,  of  course,  if  a  native 
intelligence  perspective  is  to  have  any  validity. 
Intelligence  can  be  embodied,  but  it  is  completely 
independent  of  embodiment.  For  example,  a  finite  state 
machine  (FSM)  for  a  particular  behavior  can  be  stored  on  a 
compact  disk,  located  in  the  mind  of  the  designer,  or 
spoken  out  loud  to  an  audience.  The  FSM  is  completely 
independent  of  the  medium  of  storage,  type  of 
representation,  or  mode  of  transmission.  It  exists  as 
information.  The  quantitative  measurement  of  intelligent 
systems  is  becoming  and  will  become  an  increasingly 
important  thing  to  be  able  to  do.  The  discovery  of  DNA  in 
living  things  as  the  living  blueprint  of  life  forms  an 
existence  proof  that  the  essence  of  living  things  is  non¬ 
material  information.  DNA  is  merely  the  medium;  the 
information  is  the  real  intelligence1. 

1.1  The  value  of  a  native  intelligence  metric 
A  metric  for  quantifying  the  intelligence  of  a  system 
independent  of  its  performance ,  i.e.,  a  native  intelligence 
metric  (NIM),  is  needed.  A  NIM  would  bring  substantial 
gains  beyond  that  possible  from  performance  intelligence 
metrics  alone.  A  measurement  of  native  intelligence  can  be 
made  prior  to  the  simulation  or  execution  of  the  actual 
system,  since  all  that  is  necessary  to  apply  the  metric  is  the 
representation  of  the  system  (as  a  string  of  bits),  the  rules 
for  interpreting  the  string,  and  the  meaning  matching  the 
system  (the  semantics).  To  measure  success  earlier  in  the 


1  Actually,  the  concept  that  all  hereditary  characteristics  come 
from  the  DNA  and  only  the  DNA  (called  Crick's  Central  Dogma) 
has  been  shown  to  be  false  [1], 
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design  phase  is  known  to  be  critical  to  successful 
engineering  design,  particularly  for  large  scale,  complex 
systems.  This  is  true  even  if  the  NIM  is  suboptimal, 
namely,  even  if  some  intelligence  actually  in  the  system  is 
missed  by  the  NIM.  The  ability  to  define  and  measure 
intelligence  merely  from  its  native  intelligence  offers 
promise  to  improve  important  quantities  such  as  system 
time-to-market  and  quality.  A  NIM  will  also  allow  a  more 
straightforward  debug  of  the  system,  since  the  problem  and 
solution  will  be  more  localized.  For  example,  if  we  know 
that  a  few  bits  in  the  string  cause  the  problem,  we  may  be 
able  to  easily  fix  the  problem  by  correcting  just  those  few 
bits. 

1.2  The  relationship  to  living  systems 
Our  definition  of  native  intelligence  of  artificial  systems  is 
substantially  unlike  what  is  known  as  "IQ  intelligence."  IQ 
intelligence  is  a  type  of  performance  intelligence 
(intellectual  mostly)  in  which  the  result  is  used  to  measure 
the  potential  of  the  human  subject.  Furthermore,  a 
fundamental  difference  between  living  organisms  and 
artificial  systems  is  that  the  latter  are  completely  malleable. 
This  is  not  just  a  difference  but  a  distinct  advantage. 
Therefore,  IQ  intelligence  cannot  and  should  not  constitute 
a  model  for  the  native  intelligence  of  artificial  systems. 

In  human  children  we  often  want  to  distinguish 
between  the  potential  of  a  child  to  learn  and  how  much 
they  have  actually  learned.  Being  unable  to  do  this  is  a 
source  of  incredible  frustration.  The  problem  is  that  we  do 
not  yet  understand  how  to  measure  native  intelligence  of 
humans.  I  suspect  that  some  day,  perhaps  soon,  we  will  be 
able  to  make  some  level  of  a  determination  simply  be 
interpreting  the  genetic  information  in  the  genome  of  each 
individual.  This  subject  is  the  topic  of  a  film  entitled, 
Gattaca  [2]2,  in  which  the  protagonist  overcame  his  genetic 
"destiny"  in  spite  of  an  oppressive  determinism  in  the 
surrounding  culture.  With  humans  this  is  a  loaded  topic, 
but  with  machines  the  situation  is  much  more  amenable, 
since  we  can  design  the  system  for  analysis!  In  fact,  we 
want  to  design  the  system  for  analysis.  We  want  the 
"intelligence  meter"  to  detect  the  maximum  amount  of 
intelligence,  and  since  we  have  full  control  on  how  we 
encode  the  description  of  the  design  of  the  system  (just  like 
we  can  always  encode  a  computer  program  into  a  finite 
sequence  of  bits),  we  will  choose  to  encode  it  in  a  standard 
format  so  that  the  intelligence  is  measurable. 


2  Certain  commercial  companies  and  their  equipment, 
instruments,  or  materials  are  identified  in  this  paper  in  order  to 
specify  the  experimental  procedure  adequately.  Such 
identification  is  not  intended  to  imply  any  judgement  by  the 
National  Institute  of  Standards  and  Technology  concerning  the 
companies  or  their  products,  nor  is  it  intended  to  imply  that  the 
materials  or  equipment  identified  are  necessarily  the  best 
available  for  the  purpose 


With  the  NIM  are  we  claiming  that  a  system's 
performance  is  fully  constrained  by  its  initial  determining 
information?  Isn't  this  crass  determinism?  No,  it  is  not.  A 
system's  performance  is  not  fully  constrained  by  the 
disembodied  information  describing  the  system,  since 
performance  depends  on  the  nature  of  the  environment, 
which  will  affect  future  performance.  For  example,  if  an 
adaptive  filter  adjusts  parameters  for  a  certain 
environment,  subsequent  exposure  to  another  environment 
may  cause  instabilities  in  the  filter.  High  native  intelligence 
does  not  mean  that  the  intelligence  is  realized.  The  NIM  is 
not  meant  to  exclude  the  system’s  potential  response  to 
some  future  contact  with  its  environment.  There  is  a 
multitude  of  ways  that  the  same  intelligent  system  can  by 
realized.  Finally,  living  systems  are  still  a  great  mystery 
and  are  arguably  not  fully  defined  by  algorithms  (in  the 
way  that  we  are  defining  artificial  systems)  [3], 

1.3  Specified  complexity  and  computational 
theory 

Measuring  native  intelligence  is  much  like  the  problem 
scientists  and  mathematicians  asked  around  the  turn  of  the 
century.  What  are  the  natural  classes  of  computing 
machines  and  can  we  measure  the  power  of  a  machine? 
Are  there  limits  of  computation  and  if  so  what  are  they? 
Similarly  our  question  becomes  the  following.  How  can  we 
measure  the  intelligence  of  a  system  without  ever  seeing  it 
perform  or  even  simulating  its  operation? 

In  a  manner  consistent  with  definitions  of  information 
in  standard  computational  theory  texts,  we  propose  that  the 
minimal  information  describing  a  system  entails  its 
intelligence,  i.e.,  the  intelligence  is  fully  contained  within 
the  information  describing  the  system  [4].  Of  course,  by 
information  we  do  not  mean  Shannon  information,  namely, 
that  the  information  is  merely  complex  [5].  Random  noise 
is  complex,  but  is  not  specified  {i.e.,  it  has  no  "meaning") 
and  therefore  conveys  no  real  (useful)  information.  Rather, 
what  we  mean  by  specified  complexity  is  that  the 
information  is  both  complex  and  has  discernible  meaning, 
which  is  to  say  that  it  has  recognizable  patterns  that 
indicate  the  power  of  the  system  to  perform  complex  and 
useful  tasks.  This  specified  complexity  can  be  represented 
and  analyzed  without  actually  realizing  the  system  and 
seeing  it  perform.  We  argue  that  intelligence  can  be 
distilled  down  into  pure  information,  even  a  finite  amount 
of  information.  The  description  of  a  computational 
machine  can  always  be  reduced  to  a  finite  string  of  0's  and 
l's.  To  represent  a  truly  intelligent  system,  the  system's 
information  must  be  both  complex  (in  the  Shannon  sense) 
and  specified.  In  computation  theory,  information  content 
of  a  machine  is  defined  as  the  minimum  representation  of 
that  machine.  On  the  other  hand,  certain  machines  may 
look  complex,  but  they  may  not  do  anything  useful. 
Therefore,  the  system's  description  needs  to  be  specified.  It 


needs  to  have  meaning  that,  in  some  way,  is  grounded  in 
reality  or  truth.  And  furthermore,  we  have  some  way  of 
measuring  the  amount  of  this  connection,  i.e.,  the  greater 
the  system  connects  with  real  meaning,  the  greater  the 
intelligence.  Here  then  is  a  method  for  measuring  the 
native  intelligence  of  systems:  Measure  the  information 
content,  using  information  theory  and  computation  theory 
and  measure  the  magnitude  with  which  it  can  be  matched 
to  patterns  representing  truth-grounded  semantics 
(something  like  a  dictionary  would  be  employed).  For 
example,  the  DNA  string  is  certainly  complex,  i.e.,  there  is 
no  known  simple  representation  for  the  entire  string  (no 
simple  equation  generates  the  string  of  nucleotides)  and 
DNA  is  also  certainly  associated  with  known  truth- 
grounded  semantics,  i.e.,  it  can  be  used  to  form  proteins 
that  are  the  machinery  and  computing  engines  of  the  most 
complex  factory  yet  discovered,  the  living  cell.  Clearly, 
detecting  and  measuring  the  connection  of  a  system's 
information  with  truth-grounded  semantics  is  a  substantial 
challenge. 

1.4  The  analogy  to  linear  systems  and  the 
mixed  success  of  intelligent  systems 
To  generate  an  effective  NIM,  some  mathematically  sound 
concept  of  what  fundamentally  constitutes  intelligence  is 
needed.  Why  not  use  some  of  the  existing  IS  models,  such 
as  neural  nets,  evolutionary  programming,  hierarchical 
models,  behavior-based  control,  and  fuzzy  control? 
Because  all  these  models  share  in  common  that  they  are 
primarily  models  for  facilitating  IS  design  and  execution 
and  do  not  in  themselves  constitute  formal  models  of 
intelligence  that  will  readily  yield  a  NIM. 

Why  not  use  linear  systems  theory  as  a  model  for  a 
NIM?  The  successes  of  traditional  linear  system  theory  can 
in  many  ways  be  considered  the  target  we  would  aim  for  in 
the  development  of  a  useful  NIM.  In  traditional  linear 
system  theory  we  have  a  most  agreeable  situation.  The 
dynamic  behavior  of  many  systems  can  be  described  as 
differential  equations  of  motion  and  dynamics  and, 
coupled  with  state  space  structures  and  linear  control 
theory,  we  can  determine  absolutely  whether  the  system  is 
controllable,  stable,  or  observable  without  doing  any 
simulation  or  building  and  testing  of  an  actual  system. 
However,  it  is  well  known  that  when  we  add  non-linearity 
and  time  variance,  for  example,  we  can  measure  the 
stability  of  only  a  small  subset  of  the  total  space  of  all  non¬ 
linear  systems.  The  mathematics  also  gets  substantially 
more  complicated  and  sophisticated.  A  large  class  of  real- 
world  systems  (including  discrete  event  systems)  do  not 
seem  to  submit  easily  to  concise  and  simple  mathematical 
descriptions.  So  this  path  (employing  differential  equation 
models)  for  determining  the  intelligence  of  systems  has  hit 
a  fundamental  barrier,  it  seems. 


What  about  using  formal  methods  as  a  basis  for  a 
NIM?  The  NIM  offered  in  this  paper  may  have  much  in 
common  with  formal  methods  and  further  study  of  this 
connection  should  be  pursued.  Formal  methods  have  had  a 
very  limited  impact,  but  this  may  not  be  due  to  any 
limitations  in  the  theory.  The  limited  impact  seems  to  be 
due  to  the  fact  that  many  systems  do  not  submit  easily  to 
the  formal  syntax  of  predicate  calculus  and  that  the 
learning  curve  is  too  high  for  most  system  designers  to 
comprehend  formal  methods  in  order  to  successfully  use 
them.  Tools  for  formal  methods  also  seem  to  be  lacking  or 
need  substantial  improvements  [6], 

The  field  of  intelligent  systems  research  has  rightly 
been  criticized  by  the  traditional  control  systems 
community  as  being  open  to  the  proliferation  of  ad  hoc 
design  techniques,  such  as  neural  control,  fuzzy  control, 
hierarchical  control,  etc.  These  techniques  are  not 
grounded  in  physical  and  mathematical  theories  in  the  way 
that  traditional  control  systems  are  grounded  in  the  theory 
of  dynamics  and  differential  equations.  There  is  certainly 
no  commonly  accepted  equivalent  to  differential  (or 
difference)  equations  for  large-scale  discrete-event  systems 
(large-scale  discrete-event  systems  may  be  considered  to 
be  the  domain  of  IS).  Traditional  control  systems  can  be 
measured  in  terms  of  their  stability,  controllability,  and 
observability  without  realizing  (or  embodying)  the  system. 
These  are  metrics  that  allow  us  to  measure  the  native 
intelligence  of  traditional  control  systems.  For  example,  a 
system  that  goes  unstable  in  a  certain  critical  region  would 
intuitively  be  less  intelligent  than  one  that  has  no  such 
instability  but  in  every  other  way  performs  equivalently. 
We  have  no  such  general  metrics  for  measuring  the  native 
intelligence  of  large-scale  discrete -event  systems  nor  do  we 
have  the  capability  to  compare  the  native  intelligence  of 
systems  designed  according  to  different  methods.  It  is 
important  to  have  quantitative  measures  for  the  relative 
performance  of  systems  designed  using  neural  control 
versus  those  designed  by  fuzzy  control,  etc. 

Certainly  ad  hoc  systems  are  helpful  and  even 
necessary  at  a  time  when  there  is  no  accepted  theory  with 
which  to  build  non-ad  hoc  systems.  However,  without  a 
theory,  we  often  witness  a  proliferation  of  inflated  claims 
of  system  performance  that  can  not  be  met.  This  has 
several  time  produced  a  backlash  from  funding 
organizations  who  made  financial  commitments  based  on 
these  groundless  claims.  This  is  all  the  more  reason  why  a 
well  grounded  NIM  would  be  helpful. 

Furthermore,  intelligent  systems  will  have  to  be 
designed  for  analysis.  But  this  is  exactly  what  we  do  when 
we  design  linear,  time-invariant  control  systems  in  a  state 
space  form.  It's  easier  to  analyze  that  way.  Furthermore, 
vendors  of  systems  will  not  want  to  hide  the  intelligence  of 
their  systems,  but  will  want  others  to  know  for  sure  that 
their  system  is  truly  intelligent.  The  advantage  of  having  a 


NIM  should  be  obvious.  That  way  we  can  measure 
intelligence  prior  to  any  commitment  to  simulation  in 
software  or  testing  in  hardware.  Computation  theory  can 
also  help  with  measuring  the  degree  to  which  the 
information  (describing  the  system)  matches  with  truth- 
grounded  semantics.  For  example,  a  system  that  can 
recognize  or  distinguish  between  a  broader  set  of  input 
"languages"  is  more  powerful  than  one  that  cannot.  For 
example,  a  finite  state  automata  (FSA)  can  be  designed  to 
recognize  strings  of  the  form  0*1*,  but  no  FSA  can  be  built 
that  recognizes  strings  of  the  form  0"1" ,  whereas  a  push¬ 
down  automata  (PDA)  can  be  built  to  recognize  strings  of 
either  type.  Therefore,  the  PDA  is  more  powerful 
(intelligent). 

Measuring  intelligence  is  similar  to  what  a 
cryptanalyst  does  normally  except  that  when  measuring 
intelligence  we  don't  anticipate  the  element  of  planned 
deceit.  The  cryptanalyst  is  looking  for  signs  that  the  bits 
have  identifiable  patterns  underlying  the  randomness.  If 
there  is  no  randomness,  there  are  no  patterns,  and  no 
intelligent  message  underneath.  With  randomness,  the  bits 
could  actually  be  truly  random  and  therefore  be  nonsense. 
However,  it  may  just  appear  to  be  random  and  the  patterns 
are  not  obvious  to  the  cryptanalyst.  Ostensibly,  an  IS 
would  be  designed  not  to  fool  the  IS  metrics  analyst  to 
think  the  system  is  nonsense,  but  rather  the  IS  would  be 
designed  so  that  its  intelligence  would  be  easily  perceived 
by  all. 

1.5  The  nature  of  a  NIM 

Perhaps  the  solution  is  not  in  an  analogy  to  linear  systems 
theory,  as  has  been  the  hope,  but  rather  in  information 
theory,  probability,  and  complexity  theory.  This  is  the 
research  question  we  have  asked  ourselves  and  hope  to 
provide  some  direction  towards  an  answer.  We  argue  that 
the  essence  of  intelligence  in  living  things  is  information 
(versus  physics  or  chemistry).  If  this  is  so,  we  need  to 
discover  appropriate  tools  to  correctly  analyze  that 
information  to  glean  from  it  the  level  of  intelligence  in  the 
system. 

We  claim  that  chance,  regularity,  and  intelligence  are 
mutually  exclusive.  Regularity  is  indicated  by  signals  of 
low  complexity.  Therefore  simple  Shannon  information 
filters  can  eliminate  regular  systems  from  consideration  as 
intelligent  systems  (recall  our  definition  of  native 
intelligence  as  complex  and  specified  information,  in 
which  the  level  of  complexity  and  specification  is 
proportional  to  the  level  of  intelligence).  After  first 
eliminating  regularity,  if  we  can  measure  the  probability 
that  a  system  arose  from  chance  processes  alone,  then  the 
intelligence  in  that  system  must  in  some  simple  way  be 
related  to  that  probability.  In  a  manner  similar  to  Shannon 
information  theory,  we  propose  that,  if  there  is  a  measure 
of  the  probability  that  doesn't  just  measure  the  mere 


complexity  of  the  information  (Shannon),  but  measures  the 
specified  complexity  of  the  information,  then  the 
expression  -log2  p ,  where  p  is  the  probability  that  the 
supposed  specified  complexity  (of  the  system)  arose  purely 
through  a  logical  chance  hypothesis,  is  a  theoretical 
measure  of  the  system's  native  intelligence.  Such  a  measure 
exists  [7].  We  will  describe  and  examine  this  measure  and 
suggest  how  it  applies  as  a  NIM. 

2  The  Chance  Elimination  Argument 

Our  goal  is  to  find  a  quantitative  intelligence  metric  for  an 
artificial  system  that  can  be  applied  to  the  minimal 
informational  description  of  the  system.  Another  way  of 
looking  at  this  problem  is  to  ask  ourselves,  what  is  the 
probability  that  this  system  either  arose  merely  by  chance 
(or  merely  by  regular  processes)?  Regularity  is  relatively 
easy  to  eliminate  from  consideration  since  the  information 
can  in  this  case  be  generated  from  a  relatively  small 
expression  (or  a  small  number  of  "lines  of  code").  It  may 
contain  many  bits,  but  not  have  much  information.  An 
example  of  regularity  is  an  infinitely  alternating  string  of 
four  0's  and  four  l's.  The  lines  of  code  are  simple:  write 
down  lour  0’s,  write  down  four  l's,  repeat  steps  one  and 
two.  Clearly  there  is  little  intelligence  in  this  system.  So, 
having  eliminated  regularity,  if  the  probability  that  chance 
did  the  work  is  high,  the  intelligence  in  the  system  will  be 
low;  if  the  chance  probability  is  low,  the  intelligence  will 
be  high.  Why  is  this  so?  There  are  only  three  options  for 
the  source  of  artificial  system  information:  chance, 
regularity,  or  intelligence.  It  has  to  be  one  of  the  three.  So 
if  chance  and  regularity  have  sufficiently  low  probability, 
then  the  source  must  be  intelligence.  Say  that  the  system 
needing  analysis  is  encoded  in  what  appears  to  be  a 
random  sequence.  If  we  happen  to  know  that  it  is  the 
description  of  a  complex  space  shuttle  control  system,  we 
can  readily  eliminate  chance  and  declare  it  to  be 
originating  from  intelligence.  Note  that  false  negatives  are 
possible  (thinking  it  is  a  chance  process  when  it  is  an 
intelligent  process),  but  false  positives  are  avoidable. 

A  theory  for  chance  elimination  has  been  developed 
by  Dembski  in  [7]  and  is  called  the  Generic  Chance 
Elimination  Argument  (GCEA).  This  theory  forms  the 
basis  for  the  NIM  development  in  this  paper.  Therefore, 
we  will  begin  with  a  summary  of  the  aspects  of  GCEA 
relevant  to  IS  metrics.  In  the  next  section,  we  will 
investigate  how  the  theory  might  be  (simply)  applied  to  IS 
metrics. 

We  start  with  a  subject,  S,  that  observes  an  event,  E. 
By  analyzing  the  circumstances  surrounding  E,  S  defines 
the  chance  hypothesis,  H,  gotten  from  a  reasonable  chance 
process  that  might  have  been  responsible  for  E.  S  discovers 
(doesn't  matter  how)  a  pattern,  D,  that  matches  the  event. 


E3.  With  D*  as  the  event  associated  with  the  pattern,  D,  S 
calculates  the  conditional  probability,  P(D*IH),  of  the 
event,  D*,  assuming  the  chance  hypothesis,  H,  is  true.  S 
tests  whether  P(D*IH)  is  less  than  a  universal  probability 
bound,  8 .  Dembski  determined  that  8  =  10-150  from 
fundamental  physical  limits  (space,  particles,  and  time) 
within  the  known  universe4.  Through  knowledge  of  some 
side  information,  I,  S  computes  the  conditioned  complexity 
cp( D  1 1)  of  formulating  the  pattern,  D,  given  the  side 
information,  I.  “Complexity”  in  this  context  is  defined 
within  the  domain  of  “complexity  theory,”  the  most 
common  example  of  which  is  computational  complexity. 
Complexity  theory  is  a  dual  with  probability  theory  in  the 
sense  that  with  probabilities  we  are  dealing  with  "events" 
conditioned  by  "background  information"  and  with 
complexity  theory  we  are  dealing  with  "problems" 
conditioned  by  "resources."  So  formulating  D  is  the 
"problem"  and  I  are  the  "resources"  for  solving  the 
problem.  A  problem  conditioned  by  resources  places 
<P{ D  1 1)  in  the  domain  of  complexity  theory.  S  computes  a 
tractability  bound,  A ,  which  is  used  to  calibrate  the 
complexity  values,  since  complexity,  unlike  probability,  is 
essentially  uncalibrated.  A  is  the  upper  bound  of 
complexity,  below  which  the  side  information,  I,  is 
sufficient  to  form  the  pattern,  D.  Finally,  if  P(D*IH)  <  8 
and  (p{D  1 1)  <  A  ,  S  can  infer  that  E  did  not  occur 
according  to  H.  This  is  a  substantially  abbreviated  form  of 
Dembski's  argument,  but  is  sufficient  for  our  needs.  We 
will  now  give  an  example  to  help  clarify  the  GCEA. 

Say  we  are  S  and  we  stumble  upon  Stonehenge.  We 
don't  wonder  whether  humans  carried  the  stones  (some 
weighing  over  5  x  104  kg)  to  the  site.  However,  we  do 
wonder  whether  the  particular  arrangement  of  the  stones  is 
meant  to  align  with  certain  seasonal,  celestial  events  such 
as  the  positions  of  sun  and  moon  at  summer  solstice  and 
during  eclipses.  S  declares  that  E  is  the  particular 
arrangement  of  stones  S  encounters  that  may  be  exhibiting 
alignment  with  celestial  bodies  at  certain  seasonal  times.  S 
determines  that  the  designer  of  Stonehenge  had  full  view  of 
the  sky  enough  days  in  the  year  to  be  able  to  note  seasonal 
occurrences.  S  further  determines  that  at  summer  solstice, 
for  example,  the  rising  sun  precisely  aligns  over  the  Heel 


3  Actually  if  D*  is  the  event  associated  with  the  pattern, 
D,  and  the  occurrence  of  E  implies  that  D*  also  occurred,  we  say 
that  D  delimits  E  rather  than  D  matches  E. 

4  There  are  less  than  1080  elementary  particles  in  the 
known  physical  universe,  no  more  than  1045  physical  state 
transitions  per  second,  and  (assuming  "big  bang"  cosmology)  no 
more  than  1025  seconds  of  time  will  ever  be  available  in  the 
known  physical  universe.  So  1080  X  1045  X  1045  =  10150  is  a 
liberal  measure  for  the  maximum  number  of  possible  state 
transitions  of  all  the  particles  in  the  known  universe  throughout 
the  total  lifetime  of  the  universe. 


Stone  shining  rays  directly  into  the  center  of  the  monument 
in  the  inner  horseshoe  arrangement  of  stones.  S  notes  that 
there  are  several  of  these  curious  alignments  of  lunar  and 
solar  events  (including  eclipses)  with  various  stones.  This 
alignment  of  events  with  stones  constitutes  D,  our  pattern. 
D*  then  constitutes  the  particular  pattern  of  stones 
consistent  with  the  alignments.  E  would  then,  at  the  least, 
be  delimited  by  D,  but  for  simplicity’s  sake,  say  D*  =  E. 
Now  we  calculate  the  probability,  P(D*IH),  that  D*  could 
happen  by  chance  alone,  even  though  D*  was  gotten  by 
assuming  the  matching  of  the  stones  with  particular 
celestial  events.  The  various  celestial  events  and  our 
prehistoric  Stonehenge  designer's  awareness  of  these 
events  constitute  the  side  information,  I.  We  calculate  the 
tractability  bound,  A ,  below  which  the  particularly 
alignment  of  stones  is  possible  given  I.  We  then  calculate 
^(Dll),  the  actual  complexity  of  forming  the  pattern 
given  I.  If  P(D*IH)  <  8  and  <p{\d  1 1)  <  A  ,  S  can  infer 
that  the  particular  arrangement  of  aligned  stones,  E,  did  not 
occur  according  to  the  chance  hypothesis,  H. 

All  these  definitions  and  formulations  are  completely 
consistent  with  probability  theory,  complexity  theory,  and 
statistics.  Note  that  the  measurement  is  heavily  dependent 
on  the  correctness  of  I  ( i.e .,  that  I  is  statistically 
"sufficient").  For  example,  we  must  have  sufficient 
knowledge  whether  the  prehistoric  Englishman  could 
perceive  those  celestial  objects  and  that  the  alignments 
have  changed  but  slightly  over  these  thousands  of  years. 
On  the  other  hand,  say  we  now  know  that  the  atmosphere 
of  England  was  even  more  persistently  overcast  thousands 
of  years  ago  than  it  is  today.  We  might  then  conclude  that 
tp(D  1 1)  >  A ,  i.e.,  our  prehistoric  Englishman  caught  sight 
of  the  celestial  bodies  so  infrequently  that  the  complexity 
of  forming  the  match  with  the  alignment  pattern  would  be 
virtually  impossible.  Now  a  possible  criticism  here  is  that 
our  knowledge  of  the  prehistoric  Englishman's  weather 
conditions  may  be  imprecise.  In  this  case  we  may  only  be 
able  to  determine  a  range  for  cp(p  1 1) .  All  that  would  be 
necessary  then  is  to  ensure  that  the  upper  end  of  the  range 
is  less  than  A ,  the  complexity  tractability  upper  bound. 

Furthermore,  even  if  the  complexity  happens  to  be 
tractable,  i.e.,  cp{ D  1 1)  <  A  ,  let  us  say  that  only  one  stone 
is  involved  in  the  alignment  pattern.  This  would  mean  that 
P(D*IH)  would  be  quite  high,  since  the  random  spatial 
arrangements  of  only  one  stone  are  relatively  few 
compared  to  the  random  spatial  arrangements  of  N  stones 
with  N  »  1.  Therefore,  it  is  conceivable  that  P(D*IH)  > 
8 ,  particularly  since  8  is  such  a  conservatively  small 
number,  even  though  ^£>(D  1 1)  <  A  . 

3  Chance  Elimination  Argument 

TRANSFORMED  INTO  A  NIM 


Our  goal  is  to  apply  the  GCEA  to  the  problem  of 
developing  a  NIM.  The  GCEA  is  intended  simply  to 
eliminate  the  chance  hypothesis  from  consideration, 
whereas  we  wish  to  discover  a  metric.  However,  in  the 
process  of  eliminating  chance,  the  GCEA  has  to  compute  a 
measurable  quantity,  namely,  the  probability  of  the  pattern 
(transformed  into  event  space)  conditioned  by  the  chance 
hypothesis,  P(D*IH).  The  GCEA  is  also  intended  merely 
to  compare  P(D*IH)  against  a  threshold  S ,  in  order  to 
(possibly)  eliminate  the  chance  hypothesis  and  infer 
intelligent  agency.  However,  the  artificial  systems  we 
normally  wish  to  measure  are  those  that  we  know 
intelligent  agents  have  designed  a  priori,  so  no  threshold 
comparison  is  needed.  Probabilities  are  already  calibrated, 
having  a  target  range  in  the  interval,  [0,1].  The  GCEA 
computes  a  complexity  measure,  cp(D  1 1) ,  comparing  it 
with  A ,  in  order  to  ensure  that  the  pattern  is  tractably 
constructible  from  the  side  information.  Even  though  we 
know  that  our  system  is  designed,  it  is  certainly  possible 
that  it  cannot  be  executed.  Therefore,  the  role  of  the 
complexity  measure  should  remain  the  same  for  the  IS 
measure.  Finally,  the  GCEA  computes  a  probability  and 
we  are  interested  in  a  complexity  measure.  Since  regularity 
can  be  easily  eliminated  from  consideration,  the  only  other 
options  are  chance  or  intelligent  agency.  Therefore,  to  the 
degree  our  candidate  IS  is  not  attributable  to  chance,  the  IS 
displays  intelligence.  Furthermore,  the  information 
theoretic  method  for  converting  a  probability  into  an 
information  measure  is  to  take  the  negative  of  the 
logarithm  base  2  of  the  probability.  Since  we've  defined 
information  as  the  specified  complexity  of  the  minimal 
representation  of  the  IS,  — log2P(D*IH)  is  the  NIM  we 
seek.  So  a  larger  NIM  means  higher  system  intelligence 
and  vice  versa. 

Therefore,  the  NIM  consists  the  following  steps:  1) 
identify  E  as  the  design  of  the  IS  of  interest,  2)  gather  all 
the  relevant  side  information,  I,  in  order  to  identify  the 
pattern,  D,  3)  identify  the  event  D*  that  delimits  E,  4) 
define  the  chance  hypothesis,  H,  for  the  event  D*,  5) 
compute  the  probability,  P(D*IH),  that  D*  might  occur 
according  to  H,  6)  compute  the  tractability  bound,  A ,  by 
quantifying  the  complexity  of  forming  the  pattern  with  the 
minimal  amount  of  resources,  7)  compute  the  complexity 
of  generating  the  pattern,  D,  from  I,  8)  test  if  cp{ D  1 1)  <  A , 
if  true,  then  continue  to  next  step,  or  if  false,  stop  (it  has  no 
intelligence  if  it  is  impossible  to  construct  and/or  execute), 
9)  compute  -log2  P(D*I  H) . 

How  do  we  identify  E?  This  is  simply  the  particular 
system  representation  we  wish  to  analyze.  However,  the 
system  design  needs  to  be  in  an  analyzable  format.  This 
begins  with  a  need  for  some  unambiguous  syntax  that  can 
be  represented  in  something  like  the  extended  Backus- 
Naur  form  (EBNF)  [8].  How  do  we  identify  I  and  D?  The 


relevant  semantics  will  be  captured  in  the  side  information, 
I,  and  the  gathering  and  identification  of  side  information 
is  expected  to  be  a  challenging  task.  The  challenge  should 
lie  mostly  in  the  difficulty  in  assembling  all  the  applicable 
semantic  information.  In  fact,  we  can  easily  underestimate 
the  intelligence  in  the  system  if  we  miss  or  overlook  key 
information.  Going  back  to  the  Stonehenge  example,  if  we 
mistakenly  conclude  that  the  prehistoric  man  is  inobservant 
and  therefore  is  completely  ignorant  of  eclipse  events,  but 
the  eclipse  alignments  actually  match  with  several 
alignment  of  stones,  we  will  conclude  that  the  stones 
display  less  intelligence  in  their  particular  arrangement 
than  they  actually  do.  This  is  a  false  negative,  which  will 
be  very  common.  However,  because  complex  specified 
information  is  typically  highly  differentiated  to  other 
complex  specified  information  (e.g.,  Schubert's  music  is 
very  different  from  J.S.  Bach's),  false  positives  will  be 
highly  unlikely.  The  use  of  formal  and  standard  system 
specifications  [9]  should  also  be  used  in  order  to  define 
what  is  meant  by  the  syntax.  Therefore,  intelligence  in  the 
system  will  sometimes  go  unnoticed.  Different  than  the 
Stonehenge  example,  system  designers  will  want  to  ensure 
that  the  systems  they  design  achieve  the  maximum 
(intelligence)  value  under  the  NIM.  This  is  called  design 
for  analysis  and  is  done  all  the  time  now,  though  more  for 
performance  metrics,  since  they  are  most  common. 
However,  the  current  practice  of  formal  testing  can  be 
generally  considered  in  the  same  category  as  the  NIM  [10]. 

4  Objections  to  the  NIM 

A  possible  objection  to  the  NIM  is  that  a  system  such  as  a 
neural  net  or  an  evolutionary  algorithm  starts  out  without  a 
lot  of  intelligence,  but  as  it  learns,  it  grows  in  intelligence. 
Recall  that  we  specifically  do  not  define  intelligence  as  the 
ability  to  realize  certain  tasks,  but  rather  as  the  specified 
complexity  inherent  in  the  system  at  design  time,  prior  to 
execution.  The  mere  ability  to  learn  should  be  considered 
intelligent,  independent  of,  or  prior  to,  having  learned 
anything.  The  simple  acquisition  of  knowledge,  even 
behavioristic  knowledge,  should  not  be  counted  as  an 
increase  in  intelligence.  For  example,  a  person  with 
advanced  Multiple  Sclerosis  may  be  intelligent  even 
though  he  or  she  may  neither  see,  speak,  nor  move. 

Another  objection  to  the  NIM  is  that  certain  intelligent 
systems,  particularly  learning  and  optimization  approaches 
such  as  neural  nets  or  evolutionary  algorithms  actually 
effect  an  increase  in  specified  complexity  as  the  system 
executes,  because  it  solves  complex  optimization  problems 
without  front-loaded  intelligence.  However,  it  can  be 
shown  that  through  the  designer's  choice  of  the  particular 
fitness  functions,  system  structure,  and  initial  populations, 
specified  complexity  is  entered  surreptitiously  [11].  The 
No  Free  Lunch  theorems  are  proof  of  this,  since,  for  any 
particular  optimization  algorithm,  if  the  fitness  functions 


are  not  constrained  within  a  certain  domain,  but  are 
allowed  to  freely  range  throughout  the  entire  domain  of 
possible  fitness  functions,  the  optimization  algorithm  will 
not  perform  on  the  average  any  better  than  blind  search 
[12],  So  specified  complexity  can  only  be  added  by 
intelligent  agents  [13]. 

Another  response  to  this  objection  is  largely 
experiential.  The  author  has  now  some  fifteen  years 
experience  in  the  practical  design  of  large-scale  control 
systems  for  mining  vehicles,  inspection  machines,  and  on¬ 
road  autonomous  vehicles.  The  consistently  strong 
impression  is  how  much  sophisticated,  up-front  design 
intelligence  (from  the  intelligent  agent)  is  needed  to 
generate  a  successful  system.  Furthermore,  the 
maintenance  and  improvement  of  the  system  also  requires 
enormous  input  of  human  intelligence,  very  far  from  the 
easily  and  powerfully  evolving  systems  that  are  promised. 
Perhaps  though,  we  have  been  using  the  wrong  design 
paradigm  (we  have  been  using  a  hierarchical  paradigm 
[14]).  Because  of  the  No  Free  Lunch  Theorems,  we 
suspect  that  switching  to  another  design  paradigm  will 
make  little  difference  in  this.  Furthermore,  as  a  working 
intelligent  system  designer,  we  have,  of  course,  had 
substantial  interaction  with  other  designers  using  other 
paradigms.  Based  on  this  informal  testimony,  we  have  no 
reason  to  believe  that  the  other  paradigms  require 
substantially  less  input  of  external  intelligent  agency,  both 
at  design  time  and  after. 

Yet  another  objection  is  that  performance  intelligence 
metrics  are  easier  to  obtain  and  furthermore  are  all  that  is 
needed.  Besides  the  NIM  is  too  hard  to  compute  and 
obtain,  i.e.,  too  much  work  is  required  to  make  it 
worthwhile.  This  could  be  argued  against  linear  system 
theory,  which  certainly  requires  a  substantial  mathematical 
and  physical  sophistication,  and  very  few  would  argue  that 
it  has  not  been  of  substantial  value  to  control  system 
design.  Flow  is  linear  system  theory  like  the  NIM  we 
propose?  Linear  system  theory  supplies  an  analytical 
theory  and  allows  for  metrics  for  measuring  native 
capability,  such  as  stability,  controllability,  and 
observability.  Similarly,  NIM  is  based  on  probability, 
complexity,  and  statistical  theory  and  provides  its  own 
system  capability  (or  intelligence)  metric.  As  to  the 
objection  that  the  NIM  is  too  hard  to  compute  and  obtain, 
just  in  the  same  way  that  the  effort  of  converting 
homogeneous  differential  equations  into  state  space  form 
allows  us  to  apply  metrics,  converting  our  IS  into  an 
appropriate  format,  should  also  allow  ease  of  analysis. 

One  might  argue  that  elegant  and  simple  designs 
performing  some  function  should  be  considered  more 
intelligent  than  more  complex  designs  that  perform  exactly 
the  same  function.  In  this  case  we  can  consider  as  side 
information  some  metric  like  "design  efficiency"  that  will 
increase  -log2P(D*IH)  for  the  more  elegant  design. 


Will  the  NIM  perform  well  given  a  complex  but  useful 
system  as  well  as  an  equally  complex  system  but  one  that 
simply  thrashes,  performing  no  useful  task?  The  key  here 
is  that  P(D*IH)  will  be  higher  in  the  latter  system,  since  the 
thrashing  system  is  not  specified  even  though  it  is 
complex.  The  transformed  pattern,  D*,  will  not  be 
supported  by  side  information,  I,  and  so  the  chance 
hypothesis,  H,  will  have  increased  support.  Additionally, 
how  will  the  NIM  perform  given  two  other  systems,  one 
buggy  and  one  bug-free?  Again  the  system  with  the  bugs 
will  fail  to  support  the  side  information  (that  is,  the  side 
information  affected  by  the  bugs)  that  indicates  increased 
intelligent  agency,  giving  the  chance  hypothesis  increased 
support. 

Finally,  it  will  be  argued  that  chance,  regularity,  and 
intelligence  do  not  cover  the  entire  space  of  possibilities 
and  particularly  that  intelligence  is  not  the  only  conclusion 
when  there  is  specified  complexity.  Both  Kaufmann  [15] 
and  Davies  [16]  argue  that  there  must  be  an  (as  yet) 
undiscovered  regularity  that  can  generate  specified 
complexity.  Since  this  is  not  yet  discovered  and  since  there 
is  little  concrete  evidence  for  its  existence,  it  certainly  is 
not  unreasonable  to  maintain  our  claim. 

5  Conclusions  and  future  direction 

We  make  no  claim  that  the  determination  of  the  actual 
value  of  the  NIM  will  be  easily  gotten,  particularly  as  the 
systems  under  analysis  become  more  complex.  The  precise 
determination  of  side  information,  the  matching  pattern, 
and  the  event  probability  under  the  chance  hypothesis  will 
be  challenging  to  quantify  with  reasonable  hope  of 
accuracy.  Clearly,  many  examples,  starting  with  known 
and  relatively  simple  systems  (such  as  linear  systems), 
should  be  attempted  in  order  to  exercise  this  metric. 

We  may  also  discover  that  such  a  metric  (or  any 
intelligence  metric,  for  that  matter)  cannot  measure  certain 
broad  classes  of  systems.  This  would  be  disappointing,  but 
should  not  stop  us  from  pursuing  a  metric  or  even  surprise 
us.  The  twentieth  century  has  been  known  for  the 
discovery  of  a  wide  variety  of  limitations.  Special  relativity 
posited  a  limiting  speed  for  matter.  Heisenberg  discovered 
a  fundamental  uncertainty  in  measurement  capability 
(infinite  measurement  accuracy  in  position  and  momentum 
simultaneously  of  a  particle  is  impossible).  Chaos  theory 
realized  that  the  trajectories  of  certain  deterministic 
systems  cannot  be  accurately  predicted  without  infinite 
precision  in  the  initial  conditions.  A  broad  class  of 
problems  is  either  not  computable  (like  the  general  tiling 
problem)  or  unsolvable  given  our  computing  resources 
(like  NP-hard  problems).  All  these  are  extremely  helpful 
discoveries,  even  if  somewhat  disappointing.  At  a 
minimum,  they  lessen  inflated  claims  of  performance  by 
quantifying  performance  limitations.  Perhaps  some  similar 
fundamental  limitation  for  any  intelligence  metric  is  also 


inherent  in  some  broad  class  of  artificial  systems.  To  find 
such  a  limitation  and  to  parameterize  it  would  be  a 
worthwhile  discovery. 
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